In a digital communication system, information bits (also referred to as a data stream) may be transmitted over a carrier signal by modulating the carrier signal using various modulation schemes. A known modulation scheme is QAM which maps the data stream to complex symbol constellations, such as 16 QAM, 64 QAM, and 256 QAM, for modulating a carrier signal. A demapping operation is carried by a receiver to recover the data stream.
In a hierarchical modulation, also called a layered modulation, multiple data streams are modulated into a single symbol stream. For example, two separate layers may be modulated into a single stream in which a basic layer, often called a high priority stream, may be embedded in an enhanced layer, often called a low priority stream. Receivers with good reception can receive both streams, while those with poor reception may only receive the high priority stream. Broadcasters can target two different types of receivers with different services. Typically the low priority stream offers additional information, but lower robustness than the high priority stream. For example, a broadcaster may deliver a secondary program in the low priority stream. Alternatively, the low priority stream and the high priority stream may be combined to offer high definition television (HDTV) signal.
FIG. 1 illustrates a block diagram of digital communication system 100 employing hierarchical modulation. The digital communication system comprises transmitter 104 which communicates wirelessly with receiver 108.
At transmitter 104, a first sequence of information bits {U1} is encoded by first encoder 112A. First encoder 112A yields a first sequence of coded bits {V1}. Similarly, a second sequence of information bits {U2} is encoded by second encoder 112B. Second encoder 112B yields a second sequence of coded bits {V2}. First and second encoders 112A and 112B may each be a low density parity check (LDPC) encoder.
The first and second sequence of coded bits {V1} and {V2} are fed into first and second QAM mappers 120A and 120B, respectively. QAM mappers 120A and 120B transform the bits {V1} and {V2} into QAM symbols {X1} and {X2}, respectively. The modulated symbols {X1} and {X2} are amplified by power amplifiers 124A and 124B to yield sequences of symbols √{square root over (P1)} {X1} and √{square root over (P2)} {X2}, respectively. The sequences of symbols P1 {X1} and P2{X2} are summed at summer 136 and the resulting stream may be expressed as:{X}=√{square root over (P1)}{X1}+√{square root over (P2)}{X2}
The resulting stream {X} is subsequently transmitted by transmitter 104 over the air. Assuming a channel coefficient H and noise in the channel W, the received signal at receiver 108 can be expressed by a series of QAM symbols {Y}, where{Y}=H·X+W 
The received signal {Y} is demodulated by demapper 144 to produce a demodulated stream {L(V1)} which is subsequently decoded by decoder 152 to obtain a sequence of information bits {D}. Decoder 152 may be a LDPC decoder. The demodulated stream {L(V1)} may be expressed as:
      {          L      ⁡              (                  V          ⁢                                          ⁢          1                )              }    =      log    ⁢                  P        ⁡                  [                                    V              1                        =                          0              |                              y                k                                              ]                            P        ⁡                  [                                    V              1                        =                          1              |                              y                k                                              ]                    
Traditional hierarchical modulation suffers from interference between layers, which results in both capacity loss and bit error rate increase. Also, traditional demapping applied to a hierarchical signal recovers a first layer by removing or filtering out a second layer without considering the structure of the second layer.